Using sound to extract meaning from complex data

In analyzing abstract data sets, it is useful to represent them in several alternative formats, each format bringing out different aspects of the data. While most work in data mapping has focused on visual representations, we have found that sonic representations can also be effective aids in interpreting complex data, especially when sonification is used in conjunction with visualization. We have developed prototypes for several high-level sonification tools that can be applied to a wide variety of data. While we used programmable multi-processor digital signal processing hardware to develop and experiment with these prototypes, each of these tools could be implemented as special-purpose hardware or software for use by scientists in specific applications. Our prototype tools include: Mapper (maps data to various sonic parameters), Comparator (feeds a different mapping into each speaker channel), Sonic Histogram (maps the magnitude of each category onto the amplitude of its associated sound), Shifter (shifts signals into the audible range), Marker (a sonic alarm that marks a specific condition). We tested these tools by using them to generate data-driven sound tracks for video animations generated from the same data. The resulting videos provide an increased data bandwidth and an increased sense of virtual reality to the viewer. 2. INTRODUCTION 2.1. Defining the problem Our goal is to transfer information between two sophisticated, specialized processors; one is electronic and of recent design, and the other is biological and has been undergoing strenuous testing and refinement for several million years. Modelling and analysis programs on the electronic computer are capable of generating vast amounts of data, though not necessarily in an optimal input form for the biological computer. Our job is to structure this data in ways that will improve the performance of the human observer in interpretation and analysis. There are five input ports to the biological processor: the eyes, the ears, the nose, the tongue and the skin. The first of these perceptual modes, vision, has received the most attention. We decided to investigate the auditory system and its suitability as an additional input channel. 2.2. Why use sound? There are several reasons for investigating the use of sound in data interpretation and analysis: • It adds additional bandwidth to a purely visual interface • It increases the user’s sense of engagement with the model world • People have come to expect sound to accompany animated images • The auditory system excels at certain tasks 2.2.1. Bandwidth, engagement, and expectation The auditory system can provide an alternative input channel alleviating visual overload or clutter.1,2 Visual and auditory representations can be used together to reinforce information and provide a greater sense of engagement with the model world.2,3 Our initial project was to add data-driven sound to videos of scientific visualizations generated at NCSA. In the context of watching a video, people have come to expect a sound track and may even feel uncomfortable in its absence. Rather than